Multimodal Sentence Intelligibility and the Detection of Auditory-Visual Asynchrony in Speech and Nonspeech Signals: A First Report

The ability to perceive and understand visual-only speech and the benefit experienced from having both auditory and visual signals available during speech perception tasks varies widely in the normal-hearing population. At the present time, little is known about the underlying neural mechanisms responsible for this variability or the possible relationships between multisensory speech perception abilities and performance on other perceptual or cognitive tasks. Previous studies have hypothesized that lipreading ability and auditory-visual (AV) benefit measures might be positively correlated with the ability to detect asynchronies in AV signals. Good integrators might be more attuned to detailed temporal relationships between auditory and visual information. However, this hypothesis has not been explicitly tested in normal-hearing individuals. In the present investigation, 30 normal-hearing participants were given a modified clinical test of sentence intelligibility (the CUNY sentences) under auditory-only, visual-only, and auditory-visual (AV) presentation conditions. The same participants also performed an AV asynchrony detection task using both speech and nonspeech multimodal stimuli that varied over a range of temporal asynchronies. The results suggest a relationship between auditory-only, visual-only, and AV sentence intelligibility measures and the ability to detect AV asynchrony in speech and nonspeech signals. Implications for AV integration in speech perception are discussed.